from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.212	0.0	-1	1	NaN	17.162	0.074	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.518	0.0	-1	5	NaN	0.284	0.006	0.037	0.037	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.498	0.0	1	100	NaN	15.638	0.045	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.478	0.0	-1	100	NaN	0.286	0.005	0.037	0.037	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.458	0.0	1	5	NaN	3.383	0.004	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.548	0.0	1	1	NaN	0.211	0.003	0.050	0.050	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	NaN	0.406	0.0	-1	1	NaN	3.387	0.007	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	NaN	0.419	0.0	-1	5	NaN	0.211	0.003	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.985	0.084	NaN	0.000	0.002	-1	1	0.663	0.314	0.005	6.322	6.323	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.003	NaN	0.000	0.022	-1	1	1.000	15.181	0.015	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.737	0.073	NaN	0.000	0.003	-1	5	0.757	15.254	0.012	0.179	0.179	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.003	NaN	0.000	0.023	-1	5	1.000	0.286	0.005	0.081	0.081	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.914	0.003	NaN	0.000	0.002	1	100	0.882	0.285	0.004	6.711	6.712	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	NaN	0.000	0.019	1	100	1.000	15.322	0.039	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.721	0.056	NaN	0.000	0.003	-1	100	0.882	15.258	0.019	0.178	0.178	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.003	NaN	0.000	0.023	-1	100	1.000	0.286	0.004	0.081	0.081	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.890	0.004	NaN	0.000	0.002	1	5	0.757	0.211	0.004	8.953	8.955	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	NaN	0.000	0.019	1	5	1.000	3.334	0.007	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.140	0.002	NaN	0.001	0.001	1	1	0.663	3.409	0.046	0.334	0.334	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.001	NaN	0.000	0.019	1	1	1.000	0.213	0.003	0.087	0.087	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.764	0.017	NaN	0.000	0.002	-1	1	0.896	0.211	0.003	8.347	8.348	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.002	NaN	0.000	0.004	-1	1	1.000	3.339	0.007	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.588	0.028	NaN	0.000	0.003	-1	5	0.922	3.383	0.013	0.765	0.765	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.005	NaN	0.000	0.006	-1	5	1.000	0.211	0.004	0.030	0.030	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.051	0.083	NaN	0.026	0.0	-1	1	NaN	115.662	0.000	0.026	0.026	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.844	0.041	NaN	0.021	0.0	-1	5	NaN	2.623	0.234	1.466	1.472	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.901	0.085	NaN	0.021	0.0	1	100	NaN	123.577	0.000	0.032	0.032	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.824	0.096	NaN	0.021	0.0	-1	100	NaN	2.479	0.344	1.543	1.557	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.596	0.080	NaN	0.022	0.0	1	5	NaN	0.039	0.001	93.343	93.358	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.735	0.131	NaN	0.021	0.0	1	1	NaN	0.005	0.000	716.714	716.817	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	NaN	0.021	0.0	-1	1	NaN	0.060	0.000	0.013	0.013	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	NaN	0.031	0.0	-1	5	NaN	0.005	0.000	0.102	0.102	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.852	1.103	NaN	0.000	0.001	-1	1	0.929	2.333	0.251	0.365	0.367	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	1	1.000	117.236	0.000	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.133	0.526	NaN	0.000	0.001	-1	5	0.946	127.056	0.000	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	5	1.000	2.625	0.243	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.661	0.780	NaN	0.000	0.006	1	100	0.951	2.342	0.249	2.417	2.431	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	1	100	1.000	124.242	0.000	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.238	0.356	NaN	0.000	0.003	-1	100	0.951	124.816	0.000	0.026	0.026	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	NaN	0.000	0.005	-1	100	1.000	2.332	0.233	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.758	0.540	NaN	0.000	0.002	1	5	0.946	0.004	0.000	391.830	391.940	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	NaN	0.000	0.002	1	5	1.000	0.040	0.001	0.041	0.041	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.930	0.282	NaN	0.000	0.001	1	1	0.929	0.061	0.001	15.283	15.284	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	NaN	0.000	0.001	1	1	1.000	0.005	0.000	0.191	0.191	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.027	0.023	NaN	0.001	0.000	-1	1	0.891	0.005	0.000	5.263	5.264	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.000	0.002	-1	1	1.000	0.038	0.001	0.051	0.051	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.001	NaN	0.001	0.000	-1	5	0.911	0.038	0.000	0.635	0.635	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.000	0.002	-1	5	1.000	0.005	0.000	0.383	0.384	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	92.195	0.0	300	0.001	0.001	NaN	0.45	0.022	204.668	204.909	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.102	0.001	300	0.008	0.0	0.824	0.378	0.007	0.27	0.27	See	See